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Abstract 



■ We consider a stochastic control model driven by a fractional Brownian motion. 

This model is a formal approximation to a queueing network with an ON-OFF input 
process. We study stochastic control problems associated with the long-run average 
cost, the infinite horizon discounted cost, and the finite horizon cost. In addition, we 
. find a solution to a constrained minimization problem as an application of our solution 

Q>^ I to the long-run average cost problem. We also establish Abelian limit relationships 

CN ■ among the value functions of the above control problems. 
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^ ' 1 Introduction 

Self-similarity and long-range dependence of the underlying data are two important features 
observed in the statistical analysis of high-speed communication networks in heavy traffic, 
such as local area networks (LAN) (see for instance [101 EB ESI ESI EDI EH ES] and references 
therein). In theoretical models such traffic behavior has been successfully described by 
stochastic models associated with fractional Brownian motion (fBM) (see [T^ [T5| [T9| ESJ 
[32l [33]). It is well known that fBM exhibits both of these features when the associated 
Hurst parameter is above |. Therefore, understanding the behavior and control of these 
stochastic models are of significant interest. The non-Markovian nature of the fractional 
Brownian motion makes it quite difficult to study stochastic control problems for a state 
process driven by fBM. The techniques such as dynamic programming and analysis of the 
corresponding Hamilton- Jacobi-Bellmann equations which are the commonly used tools in 
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the analysis of the stochastic control problems associated with the ordinary Brownian motion 
are not available for fBM-models. 

In this paper, we study several basic stochastic control problems for a queueing model 
with an input described by a fractional Brownian motion process. Similar queueing models, 
but not in the context of control of the state process, were considered for instance in |18l 
[23| [36] . We are aware of only a few solvable stochastic control problems in fBM setting. 
Usually, the controlled state process is a solution of a linear (semi-linear in [7]) stochastic 
differential equation driven by fBM, and the control typically effects the drift term of the 
SDE. In particular, the linear-quadratic regulator control problem is addressed [121 [HI and a 
stochastic maximum principle is developed and applied to several stochastic control problems 
in [3] . We refer to [16] and to Chapter 9 of the recent book [3] for further examples of control 
problems in this setting. In contrast to the models considered in the above references, the 
model described here is motivated by queueing applications and involves processes with state- 
constraints. At the end of this section, we discuss an example of a queueing network which 
leads to our model. Our analysis relies on a coupling of the state process with its stationary 
version (see [TH |36]) which enables us to address control problems in a non-Markovian 
setting, and our techniques are different from those employed in [3l [71 [HI [T7] . 

A real-valued stochastic process Wh = (^i?(^))j>o called fractional Brownian motion 
with Hurst parameter H G (0, 1) if Wh{0) = and Wh is a continuous zero-mean Gaussian 
process with stationary increments and covariance function given by 

Cov{Wh{s), Wnit)) = + - |t - s\^% s > 0, t > 0. 

The fractional Brownian motion is a self-similar process with index H, that is for any a > 
the process ^ (^h(c^^)) has the same distribution as {WHit))^^^. U H = ^ then Wh 
is an ordinary Brownian motion, and if if G [^,1) then the increments of the process are 
positively correlated and the process exhibits long-range dependence, which means that 

oo 

Coy{Wh{1), Wnin + 1) - Wnin)) = oo. 

n=l 

Notice that fBM is a recurrent process, and in particular, for any u > 0, lim WH{t)/t = 

t— >oo 

a.s. and consequently lim (Wnit) — ut) = — oo a.s. For additional properties and a more 

t— >oo 

detailed description of the process we refer to [221 [211 [23 [2B] • 

We consider a single server stochastic processing network having deterministic service 
process with rate > 0. For any time t > 0, the cumulative work input to the system over 
the time interval [0,t] is given by Xt + Wnit), where A is a fixed constant and Wh is a 
fractional Brownian motion with Hurst parameter H G We assume that the service 

rate fi satisfies /i > A and that the parameter fi can be controlled. The workload present 
in the system at time t > is given by X^{t) which is defined in (12. ip and (12.21) below. 
Here x > is the initial workload and M = /i — A>Ois the control variable. Assuming for 
simplicity that a; = 0, an equivalent representation for the process X" is given by (see (I2.12p 
below for the general case) 

X^{t) = {Wnit) - ut) - inf {Wnit) -ut), t>0. (1.1) 



2 



For a given arrival process Wh, this is a common formulation for a simple stochastic network 
where the server works continuously unless there are no customers in the system. The first 
term above represents the difference between the cumulative number of job arrivals and 
completed services in the time interval [0,t], and the last term ensures that the the queue- 
length is non-negative, and it is a non-decreasing process which increases only when the 
queue-length process is zero. For more examples of such formulations for queueing networks 
or stochastic processing networks, we refer to [11] and [33]. The above queueing model with 
Wh in fll.ip being a fractional Brownian motion was considered by Zeevi and Glynn in ^36j . 
In particular, we are motivated by their work. 

Our goal is to address several stochastic control problems related to the control of the 
workload process described above. The organization of the paper is as follows. We will 
conclude this section with a motivating example of a queueing network which leads to our 
model. In Section [21 we introduce the model and describe three basic stochastic control 
problems associated with it, namely the long-run average cost problem, the infinite horizon 
discounted cost problem, and the finite horizon control problem. Here we also discuss some 
properties of the reflection map which will be used in our analysis. 

In Section [3l we study the long-run average cost problem. Here we obtain an explicit 
deterministic representation of the cost functional for each control u > 0. This enables us to 
reduce the stochastic control problem to a deterministic minimization problem. We obtain 
an optimal control m* > and show its uniqueness under additional convexity assumptions 
for the associated cost functions. We show that the value function and the corresponding 
optimal control are independent of the initial data. It is well known that the above property is 
true for the classical long-run average cost problem associated with non-degenerate diffusion 
processes. Here we show that it remains valid for our model driven by fBM which is highly 
non-Markovian. The main results of this section are given in Theorems 13.41 and 13.51 Their 
generalizations are given in Theorems 13.61 and 13.71 Our proofs here rely on the use of 
a coupling of the underlying stochastic process with its stationary version introduced in 
[18] . In particular, we show that the coupling time has finite moments in Proposition 13.21 
Because of the highly non-Markovian character of the fractional Brownian motion (it is well 
known that fBM cannot be represented as a function of a finite-dimensional Markov process), 
coupling arguments in general do not work for the models associated with fBM (we refer to 
[13] for an exception). In our case, the coupling is available due to the uniqueness results 
related to the reflection map described in Section [2l 

We use our results in Section [3] to obtain an optimal strategy for a constrained optimiza- 
tion problem in Theorem 14.21 of Section m Similar stochastic control problems for systems 
driven by an ordinary Brownian motion were previously considered in [U |3T]. An inter- 
esting application of this model to wireless networks is discussed in [1]. Our constrained 
optimization problem (in the fBM setting) is a basic example of a general class of problems 
with an added bounded variation control process in the model. This class of problems has 
important applications to the control of queueing networks, but in fBM setting, it seems to 
be an unexplored area of research. 

In Section [5l we address the infinite horizon discounted cost problem associated with a 
similar cost structure. An optimal control for this problem is given in Theorem 15.31 

In Section [6l we establish Abelian limit relationships among the value functions of the 
three stochastic control problems introduced in Section [21 The main result of this section 
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is stated in Theorem 16.31 We show that the long term asymptotic for the finite horizon 
control problem and the asymptotic for the infinite horizon discounted control problem, as 
the discount factor approaches zero, share a common limit. This limit turns out to be the 
value of the long-run average cost control problem. Our proof also shows that the optimal 
control for the discounted cost problem converges to the optimal control for the long-run 
average cost control problem when the discount factor approaches zero. A similar result 
holds also for the optimal control of the finite horizon problem, as the time horizon tends 
to infinity. For a class of controlled diffusion processes analogous results were previously 
obtained in [21]. 

Motivating example. We conclude this section with a description of a queueing network 
related to the internet traffic data in which the weak limit of a suitably scaled queue-length 
process satisfies (12. ip and (12. 2p (which are reduced to the above (II. ip when the initial work- 
load X equals 0). For more details on this model we refer to [30j and Sections 7 and 8 of 
[33]. 

We begin by defining a sequence of queueing networks with state dependent arrival and 
service rates, indexed by an integer n>l and a non-negative real- valued parameter r > 0. 
For each n > 1 and r > 0, the (n, r)-th network has only one server and one buffer of 
infinite size, and the arrivals and departures from the system are given as follows. There are 
n input sources (e.g. n users connected to the server), and job requirements of each user is 
given by the so-called ON- OFF process {X^'^,i > 1} as defined in [30], namely each user 
stays connected to the server for a random ON-period of time with distribution function 
Fi, and stays off during a random OFF-period of time with distribution function F2. The 
distribution Fi is assumed to have finite mean rrii but infinite variance, and in particular. 



where 1 < < 2 for i = 1,2. While connected to the server, each user demands service 
at unit rate (sends data-packets at a unit rate to the server for processing). The server 
is processing users requests at a constant rate, say /i„,r- Assume that the ON and OFF- 
periods are all independent (for each user as well as across users), the ON-OFF processes 
have stationary increments, and average rate of arrival of jobs (packets) from each source 
(customer) is given by A = mi/ {mi + m2)- The queue-length at time t > is given by 



where L"'"^ is a non- decreasing process that starts from 0, increases only when X"''^ is zero, 
and ensures that X"'^ is always non-negative. Physically, this implies that the server is non- 
idling, i.e it serves jobs continuously as long as the buffer is non-empty. The second term in 
the right-hand side of the above equation represents cumulative number of packets sent to the 
server by all the n customers in the interval [0, t]. We will assume that X"''^(0) = Xn^ri where 
Xn,T are fixed non-negative real numbers for each n and r. In this setup, r > represents 
the time scaling parameter, and it is well known (see [29j or Theorem 7.2.5 of [33]) that 



1 - Fi{x) ~ CiX 



X"'-(t) = X"'-(0) + 5^ / xr(s)cis-/x„,.t + L'^'^(t), 




r 
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when n ^ oo first and then t ^ oo. Here Wh is a fractional Brownian motion with Hurst 
parameter H = {3 — minjai, a2})/2 G (|, l), and the convergence is the weak convergence 
in the space D([0, oo),M) of right- continuous real functions with left limits equipped with 
the standard Ja^i topology (see [33| for details). 

From the above convergence result it can be derived (see Theorem 8.7.1 in [33]) that if 
the service rates fin.r satisfy the following heavy traffic assumption 

n 

T^^n~2 (^^^ — nX) r ^ u, 
1=1 

as (n, r) oo, then a suitably scaled queue-length satisfies equations (12. ip and (12. 2p . More 
precisely, if the above heavy traffic condition is satisfied, and T~^n~2Xn,T —>■ x, then the 
scaled queue-length r~^n~2X"''^(r-) converges weakly to a limiting process that sat- 

isfies (12. ip . (12. 2p if we let n — > cxo first and then r ^ oo. 

Hence, we see that with a "super-imposed" ON-OFF input source and deterministic 
services times for the queueing processes, a suitably-scaled queue length in the limit satisfies 
our model. With a cost structure similar to either (12. 5p or (12. 7p . or (12.100 for the queueing 
network problem, one can consider the problem described in this paper as a formal fractional 
Brownian control problem (fBCP) of the corresponding control problem for the queueing 
network. We, however, do not attempt to solve the queueing control problem in this paper. 
A solution the limiting control problem provides useful insights into the queueing network 
control problem (see for instance [12]). For a broad class of queueing problems, it has been 
shown that the value function of the Brownian control problem (BCP) is a lower bound for 
the minimum cost in the queueing network control problem (see [6]). In many situations, the 
solution to the BCP can be utilized to obtain optimal strategies for the queueing network 
control problem (cf. [2], [5], [H] etc.). Here, we study just the Brownian control problem, 
which is an important problem in its own right. Our explicit solution to the fBCP can be 
considered as an "approximate solution" to the queueing network problem. 

2 Basic setup 

In this section we define the controlled state process (Section l2.ip . describe three standard 
control problems associated with it (Section 12.20 . and also discuss some basic properties of 
a reflection mapping which is involved in the deflnition of the state process (Section 12.30 . 

2.1 Model 

Let (W^//(^))j>o be a fractional Brownian motion with Hurst parameter H > 1/2 and let cr(-) 
be a deterministic continuous function defined on [0, oo) and taking positive values. For a 
given initial value x > and a control variable u > 0, the controlled state process X" is 
defined by 

X^{t)=x-ut + a{u)WH{t) + L^{t), t>0, (2.1) 
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where the process L" is given by 



= — mm 



|0, min (x — MS + a{u)WH{s))^ , 



t > 0. 



(2.2) 



The control variable u > remains fixed throughout the evolution of the state process X". 
It follows from and ([OD that X^{t) > for alH > 0. Notice that the process has 
continuous paths, and it increases at times when XJ^{t) = 0. 

The process X" represents the workload process of a single server controlled queue fed 
by a fractional Brownian motion, as described in the previous section (see also [36]). For 
a chosen control m > that remains fixed for all t > 0, the controller is faced with a 
cost structure consisting of the following three additive components during a time interval 
[t, t + dt]: 

1. a control cost h{u)dt, 

2. a state dependent holding cost C(X")(it, 

3. a penalty of pdL]^{t), if the workload in the system is empty. 

Here p > is a constant, and h and C are non- negative continuous functions satisfying the 
following basic assumptions: 

(i) The function h is defined on [0, +cxd), and 



(ii) The functions C is also defined on [0, +oo), and it is a non-negative, non-decreasing 
continuous function which satisfies the following polynomial growth condition: 



for some positive constants K > and 7 > 0. 

We will sometimes assume convexity of h and C in order to obtain sharper results, such as 
the uniqueness of the optimal controls. 

2.2 Three control problems 

Here we formulate three cost minimization problems for our model. In the long-run average 
cost minimization problem (it is also called ergodic control problem), the controller minimizes 
the cost functional 



h is non-decreasing and continuous, h{0) > 




(2.3) 




(2.4) 




(2.5) 
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subject to the constraint u > for a fixed initial value x > 0. Notice that since L^{t) is an 
increasing process, the integral with respect to L'^{t) can be defined an ordinary Riemann- 
Stieltjes integral. The value function of this problem is given by 

Vo(x) = inf (2.6) 

u>0 

In Section [3], we show that I{u,x) and hence also the value function Vo{x) are actually 
independent of x. In addition, we show the existence of a finite optimal control m* > and 
also prove that u* is unique if the functions h and C are convex. We apply the results on 
the long run average cost problem to find an optimal strategy for a constrained optimization 
problem, in Section HI 

In Section [5l we solve the infinite horizon discounted cost minimization problem for the 
case when cr{u) = 1 in (12.11) . In this problem it is assumed that the controller wants to 
minimize the cost functional 

Ja{u,x) := e(^J^ e-'''[h{u) + C{X^{t))]dt + p e-'^'dLlit)). (2.7) 

subject to u > for a fixed initial value x > 0. Here the discount factor a > is a strictly 
positive constant. The value function in this case is given by 

\4(a;) = inf J„(u,x). (2.8) 

We study the asymptotic behavior of this model in Section [61 When a approaches zero, 
we prove that lim aJaiu^x) = I(u,x) for any control m > 0. Furthermore, we show that 

lim aVa(x) = Vo(x) and the optimal controls for the discounted cost problem converges to 

a-vO+ 

that of the long-run average cost problem as a tends to zero. In Section [6l we also consider 
the finite horizon control problem with the value function V{x, T) defined by 



V{x,T):=miI{u,x,T), (2.9) 

u>0 



where 

/(m,x,T) 



: [Mn) + C(X:(t))]dt + pL^(T)^ 

= hiu)T + pE{L:{T))+E(^£c{X:{t))dt), (2.10) 

and p > is a non- negative constant. We prove that lim ^^^'"^^ = Vq{x). Furthermore, 

we show that the optimal controls for the finite horizon problem converges to that of the 
long-run average cost problem, as T tends to infinity. 



2.3 The reflection map 

The model equations (12.11) and (12. 2p have an equivalent representation which is given below 
in (I2.12P by using the reflection map. Therefore we briefly discuss some basic properties of 
the reflection map and of the representation (12.12^ . 
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Let C([0, oo),R) be the space of continuous functions with domain [0, oo). The standard 
reflection mapping T : C([0, oo), M) ^ C([0, oo), M) is defined by 

r(/)(t) = /(t)+ sup(-/(s))+, (2.11) 

s€[0,t] 

for / G C([0, oo), M). Here and henceforth we use the notation a"*" := max{0, a}. This mapping 
is also known as the Skorokhod map or the regulator map in different contexts. For a detailed 
discussion we refer to j2ni [33] • 

In our model (I2.ip - (l2.2p . we can write as follows: 

X^{t) = r{x-ue + a{u)WH){t), (2.12) 

where e(t) := t for t > 0, is the identity map. 

Note that according to the definition, r(/) (t) > for t > 0. We will also use the following 
two standard facts about F (see for instance [20l[33]). First, we have 

sup |F(/)(t)| <2 sup |/(t)|, (2.13) 

iG[0,T] t£[0,T] 

for / G C([0, oo), M). Secondly, let / and g be two functions in C([0,oo),]R) such that 
/(O) = g{0) and h{t) := f{t) —g{t) is a non-negative non-decreasing function in C([0, oo), M). 
Then 

r(/)(t)>F((7)(t), foralH>0. (2.14) 

We shall also rely on the following convexity property of the reflected mapping. Let a G (0,1) 
and / and g be two functions in C([0, oo), M). Then, af + {1 — a)g G C([0, oo), M), and 

T{af + (1 - a)g){t) < aV{f){t) + (1 - a)V{g){t) (2.15) 

for all t > 0. The proof is straightforward. Let F{x) = x~ := max{0, —x}. Then F is 
a convex function and therefore (a/(s) + (1 — a)g{s))^ < af~{s) + (1 — a)g^{s). Conse- 
quently, sup,g[o,t](a/(s) + (1 - a)g{s))- < « sup,g[o£/'(s) + (1 - a) sup.gfo,^] ^~(s)^_Since 
suPse[o,t](~/('5))^ = sup^gjg^j] f^{s), the inequality (12.151) follows from the definition (12.111) . 

The reflection map also satisfies the following minimality property: if ?7 G C([0, oo), M) 
are such that ip is non-negative, r]{0) = 0, 77 is non-decreasing, and iplt) = ip{t) + ri{t) for 
t > 0, then 

ip{t) >r{<^){t) and r]{t) > sup {-i^{s)y , for alH > 0. (2.16) 

se[o,t] 

3 Long-run average cost minimization problem 

In this section we address the control problem defined in fl2.5p - fl2.6l) . First we find a solution 
to the control problem for the particular case when (t{u) = 1 in (12.10 . This is accomplished 
in Sections 13. 1113.41 In Section 13. 5[ we show that the general case can be reduced to this 
simplified version. 
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3.1 Reduction of the cost structure 

The controlled state space process X" (corresponding to a{u) = 1) has the form 

X^{t) = x-ut + WH{t) + Ll{t), t>0. (3.1) 

The following lemma simplifies the expression for the cost functional fl2.5p by computing 
^hm 1E(L«(T)). 

Lemma 3.1. Let be given by ([31]). Then lim ^E{L^{T)) = u. 
Prooj. Since m > 0, using (l2l2D . fl213|) . and flCTj) . we obtain: 

< X^(t) < X°(t) =r(x + Vrj,)(t) <2(|x|+ sup \Wh{s)\). 

sG[0,t] 

By the self-similarity of fractional Brownian motion process, 

E[ sup \Wu{s)^ <K{r'', 
se[o,T] 

where G (0, oo) is a constant independent of T (see for instance [211 P- 296]). Therefore, 

< E(X;^(T)) < 2Ki(|x| +r^). (3.2) 
Consequently, lim \E{XI{T)) = 0. Since Wj/(r) is a mean- zero Gaussian process, it follows 

from ([SID that \E[LI{T)) - u = ^(^E{X^{T)) - x). Letting T tend to infinity completes 
the proof of the lemma. □ 

Remark. Lemma \3.1\ with literally the same proof as above remains valid if satisfies 
( CT instead of CT . 

With the above lemma in hand, we can represent the cost functional (12.51) and reformulate 
the long-run average cost minimization problem as follows. The controller minimizes 

I{u,x) = {h{u) +pu) +limsup;^E( [ C(X^(t))rft) (3.3) 

subject to u > and X^ given in (13.11) . Note that the above reduction shows that the original 
minimization problem (12. 5p reduces to the case p = with the function h{u) replaced by 
h{u) + pu. 

Our next step is to analyze the cost component limsup Tiri?! /J C(X"(t))(it j . The fol- 

lowing results are described in [18] and [8] , and are collected in [36] in a convenient form for 
our application. We summarize them here using our notation. 

(i) The random sequence Xq (t) with t > converges weakly, as t goes to infinity, to the 
random variable 

:= max{iyff(s) - us). (3.4) 

s>0 
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(ii) There is a probability space supporting the processes Xq , Lq (and hence X" as well as 

for any x > 0) and a stationary process X* = {X*{t) : t > 0} such that 

Kit) = Wnit) -ut + max{X:(0), t > 0, (3.5) 

and 

X:(t) = Z„, t> 0, (3.6) 

where "=" denotes the equality in distribution and Zu is defined in (13.40 . 

(iii) The tail of the stationary distribution satisfies 



hm 3^^-^ logP(Z„ > 3) = -e*iu), (3.7) 

3— >oo 



where 6*{u) is given by 



r(u) = TT-^ > 0. (3.8) 

In particular, all the moments of are finite. 

Throughout the rest of the paper, we use this probability space where all these processes are 
defined. Using (13.1 p and (12. 2p . we can write for t > 0, 

X^{t) = Wnit) -ut + max{x, L^{t)}, (3.9) 

where 

L^{t) = - inf {Wh{s) - us) = sup {us - Wh{s)). (3.10) 

*e[o.*] se[o,t] 

The above representation (13. 9p and (I3.10p agrees with (13.50 if the process X" is initialized 
with X*(0). 

3.2 A coupling time 

The following coupling argument is crucial to address the optimal control problems. In 
particular, it enables us to deal with the last term of I{u, x) in (13. 3p . 

Proposition 3.2. Let u > and the initial point x > be fixed. Consider the state process 
X^ in (13. ip and the stationary process X* of (13.50 and (13. 6p . Then the following results 
hold: 

(i) There is a finite stopping time Tq such that X"(t) = X*(t) for all t > Tq. Furthermore, 
E{t^) < oo for all (3 > 0. 

(ii) The cost functional I{u,x) defined in (12.50 is finite and independent of x, that is 
I{u,x) = l{u,0) < oo for a; > 0. Consequently, the value function Vq{x) = inf I{u,x) 

is also finite and independent of x, that is Vo{x) = Vo(0) < oo for x > 0. 
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Proof. For y > introduce the stopping time 

Aj, = inf{t > : L'^{t) > x + y}. 

The stopping time Xy is finite a.s. because hm L]^(t) > hm (ut — Wnit)) = +00 a.s. 
Define the stopping time tq by 

To = inf{t > : L^{t) >x + X^iO)}. (3.11) 

Here X* is the stationary process which satisfies (13.51) and (13.61) . That is tq = Xx*{o) a-s. It 
follows that for t > tq, we have L^{t) > L^{to) = x + X*{0) and X*(0) > 0. Therefore, it 
follows from ((331) and (13A0|) that X^{t) = Wnit) - ut + Ll{t) = X*{t) for t > tq. 

Next, we prove that E{tq) < +00 for each /? > 0. Clearly, without loss of generality we 
can assume that /? > 1. We then have: 



e{t,^) <Y,E{\t^, ■ Mn.<xm<n.+i]) < Y.[E^^Zi)nxm > m)Y^\ (3.12) 



1 1/2 

m=0 m=0 



where in the last step we have used the Cauchy-Schwartz inequality. Since X*{0) has the 
same distribution as = sup5>o{W^//(s) — us}, it follows from (13.71) that for all m large 
enough, 

P(X:(0) > m) < (3.I3) 

where 6*{u) is defined in (13. 8p . Next, we estimate E{\^) for m > and j3 >1. For m G N, 
let hm = X + m and = We have 

^(A^^) = 2/5 / t^^-^P{\m > t)dt = 2(3 t^l^-^P{Ll{t) <x + m)dt 
Jo Jo 

POO 

< Tl^ + 2(3 eP-^P{Ll{t) < bjdt, (3.14) 

where the second equality is due to the fact that P{Xm > t) = P{Lo{t) < x + m) according 
to the definition of A^. Notice that 

P{L^{t) < bj) = P{ sup {us - Wh{s)} < hm) < P{WH{t) >ut- bj), (3.15) 

se[o,i] 

and recall that Z := ^"n^^ has a standard normal distribution. Therefore, by (13.151) . for 
t > Tm we have 

P{E^oit) < bn) < P{WH{t) >Ut- bJl < P{WH{t) > y) 

= p(z>^^^). (3.16) 
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It follows from (IXTD and (KWh that 

POO 



ut 



l-H 



2\Z\ 



u 



2<3 
1-H 



4/3 



> t 



u 



213 



dt 



(x + mf^ + E 



2\Z\ 



u 



2/3 
1-H 



< cx). (3.17) 



The estimates fl3.13p and f l3.17p imply that the infinite series in the right-hand side of fl3.12p 
converges. Thus E{tq) < oo for all /3 > 1, and hence for all /3 > 0. This completes the proof 
of the first part of the proposition. 

We turn now to the proof of part (ii). First, we will prove that 



E(^j^\c{x:{t))-c{x:{t))\dt) < oo. 



(3.18) 



We will show later that part (ii) of the proposition is a rather direct consequence of this 
inequality. 
Notice that 

£;(2°°|c(x,"(t)) -c(x:(t))|dt)=E(^"|c(x:(t)) -c(x:(t))|dt), 

where tq is given in 03.111) and X* is given in (13.51) and 03.61) . The definition of tq implies 
^0 (^o) <x + Xl{{}). Therefore, it follows from ([33D and ([33]) that for t G [0, tq], 

max{X:(t), X:(t)} <Z^ + x + Xm. 

Since C is a non- decreasing function, this implies 

max{C(x:(t)),C(x:(t))}<C(Z. + x + X:(0)), 

and consequently, using the Cauchy-Schwartz inequality, 

E(^j^J\C{X:{t))-C{X:{t))\dt) < E(roC{Z^ + x + X:{0)) 



< E{t',)E[[C{Z^ + x + X:{0))] 
Since E{tq) < oo by part (i) of the lemma, 03.181) will follow once we show that 

E([C(Z„ + a; + X:(0))]') < oo. 



1/2 



(3.19) 



Recall that X*(0) and Zu have the same distribution. The tail asymptotic 03.71) implies that 
any moment of Z^ is finite. This fact together with 02. 4p yield 03.191) . 
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We will now deduce part (ii) of the proposition from fl3.18p . Toward this end, first observe 
that, since X*{t) is a stationary process, 

^E[j^^ C{X:{t))dt) = ^E(^£c{X:{0))dt) = E{C{Z^)), 

and recall that E(^C{Zu)) < oo by (13.71) . Then notice that by (13.181) . 

\imsnp\^E(^£ C{X:{t))dt) - ^e(^£ C{X:{t))dt) 

< limsupiE( r\C{x:{t)) - C{x:{t))\dt) = 0. 

Therefore 

hm ^E{ r C{X:{t))dt) = E{C{Z^)), (3.20) 

which implies part (ii) of the proposition in view of (13. 3p . Therefore, the proof of the 
proposition is complete. □ 

Remark. The above proposition is in agreement with a result in Theorem 1 of which 
shows that the asymptotic distributions, as t approaches infinity, for M{t) = maXse[o,t] X'^{s) 
and M*{t) = max^gp^t] X*(s) coincide. 

3.3 Properties of E(C(Z„)). 

For M > 0, let G{u) = E(C(Z„)). We are interested in the behavior of G{u) in view of the 
identity (^M>- 

Lemma 3.3. Let G{u) = E(^G{Zu)) where Z^ is defined in (13. 4p . Then the following results 
hold: 

(i) G{u) is a decreasing and continuous function of u on [0, oo). 

(ii) IfG{x) is a convex function then so is G{u). 
(Hi) lim G(u) = +oo. 

Proof. First we observe that the polynomial bound (12. 4p on the growth of G combined with 
(13. 7p . which describes the tail behavior of imply that G{u) = i?(C(Z„)) is finite for 
each M > 0. It is a decreasing function of u because G is non-decreasing while Zu^ < Z^^ if 

To complete the proof of part (i), it remains to show that G{u) is a continuous function. 
To prove this, first notice that, according to the definition (13.41) . Zu is a continuous function of 
the variable U cl. S. , cLS shown below. Indeed, if tu is a random time such that Zu = WH{tu)—utu 
and u G (0, v), then 

Zu> Z^> WHity) - vtu = Zu- tu{v - u). 
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Hence lim^^„+ Z^, = A similar argument shows that hmt,^„- = Zu- Therefore, since 
C is a continuous function, C{Zu) is a continuous function of u with probabihty one. Since 
C{Zu) is monotone in u, the dominated convergence theorem imphes the continuity of G. 

To prove part (ii), fix constants r G [0, 1], Ui > 0, U2 > 0, and let Ur = rui + (1 — r)u2. 
Then Zu^ = supi>o{W^H(i) — Urt} < rZu^ + (1 — r)Zu2- If C is a non- decreasing convex 
function, we have~E(C(ZsJ) < rE{C{Zu,)) + (1 - r)E{C{Zu,)). Hence G is convex, and 
the proof of part (ii) is complete. 

Turning to the part (iii), we first notice that Zq = sup^^Q Wnit) = +00 with proba- 
bility one. Let (m„)„>o be any sequence monotonically decreasing to zero. Then, Zu„ is 
increasing and hence there exists a limit (finite or infinite) lim Zu„ = L and Z^^ < L for 

n— >oo 

all n > 0. Thus Wnit) — Unt < L for all n > and t > 0. By letting n go to infinity 
we obtain supj>oW//(t) < L, and consequently L = +00 with probability one. Therefore 
lim Z(u) = +00 a.s. Since C is a non- decreasing function, the monotone convergence 

theorem implies that lim E(^C{Zu)) = +00. This completes the proof of the lemma. □ 
3.4 Existence of an optimal control 

In the following two theorems we provide a representation of the cost functional I{u, x) as 
well as the existence and uniqueness results for the optimal control m* > 0. 

Theorem 3.4. Let I{u,x) be the cost functional of the long-run average cost problem de- 
scribed in (13.31) . Then 

(i) I{u,x) is independent of x and has the representation 

I{u) := I{u, x) = h{u) +pu + G{u), (3.21) 

where G{u) is given in Lemma \3.3[ Furthermore, I{u) is finite for each m > and is 
continuous in u > 0. 

(ii) lim I{u) = +00 and lim I{u) = +00. 

(iii) If h{x) and C{x) are convex functions, then I{u) is also convex. 

Proof. Part (i) follows from fl3.3p . Proposition 13.21 and Lemma [3.31 

The first part of claim (ii) follows from the fact that I{u) > G{u) along with part (iii) 
of Lemma 13. 3[ To verify the second part, notice that I{u) > h{u) for all m > 0, and 
lim h{u) = -\-oo. Consequently, lim I{u) = +00. 

«— > + 00 M— » + 00 

Part (iii) follows from the representation fl3.2ip combined with the part (ii) of Lemma l3.3[ 
This completes the proof of the theorem. □ 

Theorem 3.5. 

(i) There is an optimal control u* > such that for all x > we have 

I{u*) = min/('U, x) 

M>0 

where I is given in (I3.2ip . In particular, u* is independent of x. 
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(a) In the case p > 0, if h and C are convex functions, then u* is unique. 

(Hi) In the case p = 0, if h is a strictly convex function and C is a convex function, then u* 
is unique. 

Proof. Since I{u) is a continuous function, part (i) follows from parts (i) and (ii) of Theo- 
rem I3.4[ 

If h and C are convex functions, the representation fl3.2ip yields that J is a strictly convex 
function when p > 0. Therefore, u* is unique in this case. 

In the case p = 0, if /i is a strictly convex function and C is a convex function, the results 
follows from the representation (13.211) in a similar way as in the case p > 0. □ 

3.5 Generalizations 

In this section we generalize the results in Theorem 13.41 and 13.51 to the more general model 
introduced in (12.11) . Notice that for a given control m > fixed in (12. ip . the self-similarity of 
fBM yields that the process Wr defined by 



WH{t)=a{u)WH(—-T), te 
is a fractional Brownian motion. Let 



Then Yl" satisfies 



ut — - 

y:it) = ^ - — -X + Wnit) + Ll{t), (3.22) 
a[u) H 



where 



Using fl2.2l) and change of the variable s = — ^-, we observe that 

M V o ^^^^^ 



Llit) = ^"(-V) = -minjo, min (a: - + Wh{s))]. (3.23) 



The equations (I3.22p and (I3.23P are analogous to (12.11) and (12. 2p . 

We next consider the change in the cost structure due to the change of the variable 
s = — We notice that 

r-M{T) 



f I c{x:{t))dt = ^ c{Y:{t))dt, 
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where M{T) = a{u)HT. Therefore, using the results in Theorem 13.41 we obtain 

hm 1-e( f C{X:{t))dt) = g(-^). (3.24) 



fJ I 

We have the following result. 



Theorem 3.6. Consider the controlled state process X" defined by (12. ip and (12. 2p with the 
cost functional I {x,u) given in (12. 5p . Define f : [0, oo) [0, oo) by 

m = — -X- (3.25) 

a{u)H 

Then 

I{u,x) = h{u) +pu + G{f{u)), (3.26) 

where the function G is given in Lemma \3. 3[ Furthermore, I{u,x) is independent of x (we 
will henceforth denote the cost function by I{u) ). 

Proof. The same argument as in the proof of Lemma [3. II yields ^im j;E(^L^(T)^ = u. Com- 
bining this result with (13.241) we obtain the representation (13.261) . □ 

Our next result is analogous to Theorems 13.41 and 13. 5[ 
Theorem 3.7. Assume that the function f in (13.251) is continuous and lim fiu) = 0. Then, 
with I(u) = I{u,x) as in (I3.26p . we have: 
(i) lim I{u) = lim I{u) = +oo, and I{u) is a finite continuous function on [0,oo). 
Furthermore, there is a constant u* > such that I{u*) = minuyo I{u). 

(a) If f is a concave increasing function then the statements similar to parts (i) and (ii) 
of Theorem \3.5\ (regarding the uniqueness u*) hold. 



The proof of this theorem is a straightforward modification of the proofs of Theorems [X 
and 13.51 and therefore is omitted. 

Remark. One can further generalize our model to cover the following situation. For given 
positive continuous functions b{u) and a{u) let 

X:{t) = X + a{u)Wu{t) - b{u)t + Llit), 

where for m > 0, 

L"(t) = -min |o, min (x - b{u)s + a{u)WH{s)) 

The optimization problem here is to minimize the cost functional I{u,x) defined in (12. 5p . 

Following the time change method described in Section l375\ one can obtain an analogue of 
Theorem 3. 7 regarding the derivation of the optimal control. In this situation, the function 
f defined in (I3.25P needs to be replaced by f(u) = b{u){a{u))~H with the assumptions that 
f is continuous and lim f(u) = 0. We omit the details of the proof. 
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4 A constrained minimization problem 



In this section, we address a constrained minimization problem that can be solved by using 
our results in Section [31 Our model here is of the form 

Y^^{t) =x-ut + a{u)WHit) + (4.1) 

where a is a non-negative continuous function, K'^{-) is a non-negative non-decreasing right- 
continuous with left limits (ROLL) process adapted to the natural filtration {J-'t)t>o, where 
J^t is the (T-algebra generated by {Wh{s) : < s < t} augmented with all the null sets. 
Furthermore, K'^{0) = and the process K"^ is chosen by the controller in such a way that 
the state process is constrained to non-negative reals. In this situation, the controller 
is equipped with two controls: the choice of m > and the choice of K"^ process subject 
non-negativity of the process. 

Throughout this section we keep the initial state x > fixed. We will deduce using the 
results of the previous section that the value of this minimization problem as well as the 
optimal control are not affected by the initial data. 

Let m > be any fixed positive constant. The constrained minimization problem we 
would like to address here is the following: 

Minimize 

\imsnp^E([ [/i(M) + C(F"(t))]rft) (4.2) 

Subject to: 

limsup < m. (4-3) 

A controlled optimization problem of this nature for diffusion processes was considered in 
[1], and for a more complete treatment in the case of diffusion processes we refer to |31j . 
Fix any integer m > and define a class of state processes Um as follows: 

U„, = \ (r", K") : r"(t) > for t > 0, dHD is satisfied, limsup ^ ^ < m 

From our results in Section [3] it follows that for any u < m, the pair (X^,L^) in (12.11) and 
(12. 2p belongs to Um, and hence Um is non-empty. Therefore, the constrained minimization 
problem is to find 



inf limsup— -El / 

In this section we make the following additional assumptions: 
(i) For functions h and C we assume: 



h is strictly convex and satisfies (12. 3p . C is convex and satisfies (12. 4p . (4.4) 
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(ii) Let f{u) = Then 

cr{u) TT 

f(u) > 0, lim f(u) = 0, and / is a convex increasing function. (4.5) 

The following lemma enables us to reduce the set Um to the collection of processes (X", L") 
described in the previous section, with u <m. 

Lemma 4.1. Let m > and let (F", JC") he a pair of processes satisfying ( 14. ID . Consider 
(X", L^) which satisfies (12.11) . (12. 2p . and is defined on the same probability space as (y , K^). 
Then 

(i) mt) < K'^it) and X^{t) < F"(t) fort>0. 
(li) u = hm ^E{Ll{T)) < limsup ^E{K''{T)). 



(m) G(/(«)) = hm iE(/,^C(X«(t))rft) < limsup iE(/;C(r«(t))rft). 

1 ^oo T->oo 

Proof. Since, Y"^ > 0, K"(0) = 0, and is non-increasing process, the minimality property 
of the reflection map stated in (12161) implies that L^{t) < K'^it) and X^{t) < Y^{t) for 
t > 0. 

Next, observe that part (ii) of the lemma follows from the result in part (i) while the 
identity u = lim ^E(L'^{T)) is implied by Lemma [3.11 (see also the remark right after the 

proof of Lemma 13. ip . 

Finally, part (iii) of the lemma follows from (13.240 . part (i), and from the fact that C is 
non-decreasing. The proof of the lemma is complete. □ 

Let 

Vm = {(X^, L^) : ([2II]),([221) are satisfied and in addition u<m}. 
From the above lemma it is clear that 

inf lim [ \h{u) + C(X^it))]dt) 

= rx4Cv ^^^^^P^E(r[hiu) + C{Y-{t))]dt). (4.6) 



Therefore, our minimization problem is reduced. Next, we can use the results in Section [3? 
and write for any m > 0, 

^im r[Mt.) + C(X:(t))]rft) =M«) + G(/(tz)). (4.7) 

J 

Here G is given by Lemma 13. 3[ and / is described in (I3.25P and (14. 5 p . Consequently, 

inf hm ]^E{l\h{u)^C[Xl{t))\dt\ 

= inf{/i(M) + G(/(m)) : < M < m}. (4.8) 
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We next consider the optimal control described in Theorem 13.71 corresponding to the case 
p = 0. In virtue of the assumptions (14. 4p and (14. 5p . the optimal control is unique and we will 
label it by > 0. We have the following result. 

Theorem 4.2. Let 

u*(m) = l "'^"O' (4 9) 

^ ' \u* if m > ' ^ 

where Mq is the unique optimal control in Theorem \3. 7| corresponding to p = 0. 

Then the pair {Xx ^"^\Lx ^'^^) is an optimal process for the constrained minimization 
problem (14. 2 p and (14. 3p . Furthermore, the optimal control u* {m) is a continuous increasing 
function of the parameter m. 

Proof. Let A(m) = h{u) + (?(/(«)) where / and G are as in (14.70 . Then, by the assumptions 
(14. 4p , (14. 5 p and Theorem 13. 7[ A is a strictly convex function which is finite everywhere on 
(0, oo). Furthermore, lim A{u) = +oo and lim A(m) = +oo, and hence A has a unique 

minimum at Uq. Therefore, A is strictly increasing on (mq,oo). Clearly, with u*{m) defined 
in dS]), 

A(u*{m)) = inf A(m), 
and u*{m) is the unique number which has this property. By (14. 6 p and (14.81) . we have 

Consider the pair of processes [X^ Lx ^"^^) defined in (12. ip and (12. 2p . Then, in virtue of 
Lemma EH we have lim ^E(L''*(™)(T)) = u*{m) < m, and by Theorem 3.6, 

hmsupi^f r[h{u) + C{Xf^"^\t))]dt) = A{u*{m)). 

Hence (X^ ^'"^Lx ^'"'*) describes an optimal strategy. This completes the proof of the theo- 
rem. □ 

Remark. Notice that the above optimal control u*{m) is independent of the initial point x. 



5 Infinite horizon discounted cost minimization prob- 
lem 

In this section we define an optimal control u* for the infinite horizon discounted cost func- 
tional given in (12.70 . Throughout this section we assume that a{u) = 1, the state process X^ 
satisfies (13. ip . and that the functionals h and C are convex in addition to the assumptions 
stated in (12. 3p and (12. 4p . In contrast with Section [3l our methods here do not readily extend 
to the case where the function cr(u) is non constant. 
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h{u) 



The discounted cost functional Ja{x,u) is given by 

Ja{x,u) = e(^J^ e-'''[h{u) + C{X^{t))]dt + e-'^'pdL^it) 

+ E(^l\--'[C{X:{t))+apL:{t)]dt). (5.1) 

Here a > is a constant discount function. To derive the last equality above we have used 
Fubini's theorem to obtain e~"^dL^{t) = a e~"^L^{t)dt. 
Let 

Ja,iix,u) = E(^j\-'^'C{X':{t))dt) 

and 

J^,2{x,u) = e(^J^ e-'''Ll{t)dty (5.2) 

Next we use the convexity of the reflection mapping described in fl2.15p to establish the 
convexity of the cost functional with respect to u. 

Lemma 5.1. Let x > be fixed and let C be a convex function satisfying assumption (12 ■4p . 
Then, Ja^i{x,u) and Ja,2{x,u) introduced above are finite for each u>0 and are convex in 
u-variable. 

Proof. By (O, we have the bound E[L^{t)) < ut + Ko{l + t^), where i^o > is a constant 
independent oft. This implies, by using Fubini's theorem, that Ja,2{x,u) is finite. 
Next, by using (13.181) we obtain 

J„,i(x,n)-E(^"e-"*C(X:(t))rft)| <i?(^~|C(X:(t)) -C(X:(t))|rft) < oo. 

But, using the stationary of X*, we have eQ^ e-'^^C {X*{t))dt^ = a-^E{C{Zu)), where 

Zu is given in (13. 4p . Notice that E(^C{Zu)) is finite because C has polynomial growth and 
in virtue of (13. 7p . Consequently, Ja,i{x,u) is also finite. 

To establish convexity of Ja,i{x,u), first recall that X" = T(^x + Wh — ue), where e(t) = t 
for t > 0, and F is the reflection mapping described in Section [231 Now let Ui >0,U2> 0, 
and r G (0, 1). Then, for t > 0, 

X + Wnit) - {rui + (1 - r)u2)e{t) 

= r{x + Wnit) - uie{t)) + (1 - r) (x + Wnit) - U2e{t)). 

Since the reflection map F satisfies the convexity property (12.150 . we have: 

(t) < rX^^it) + (1 - r)X:^(t), (5.3) 

where Ur = rui + (1 — r)u2. Next, since C is a convex non-decreasing function, (15. 3p implies 
that C{X^-{t)) < rC(X^i(t)) + (1 - r)C{X^^{t)) for t > 0. From this it follows that 

Ja,lix,Ur) < rJa,lix,Ui) + (1 - t) J^^i^X , U2) ■ 
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Hence Ja,i{x,u) is convex in -u- variable. 
Next, by and fl2ni|) . we have 

T{x + Wh- ue){t) -{x + Wh- ue){t) = Ll{t), t > 0. 

Then, since F is convex in m- variable by (12.151) . the process L"^ is also convex in u- variable. 
Consequently, with Ur = rui + (1 — r)u2, we obtain 

L^^(t)<rL-(t) + (l-r)L-(t), t > 0. 

Finally, it is evident that Ja,2{x,u) is convex in m- variable from the definition (15. 2p . □ 

Corollary 5.2. Under the conditions of Lemma \5.1\ the discounted cost functional Ja{x,u) 
is finite for each u > and is convex in u-variable. 



Proof. Notice that 



h(u) 

Ja{x^ U) = h Ja l[x, U) + apJa 2[x , u) . (5.4) 

a 



By our assumptions, /i is a convex function and p > and a > are constants. Therefore, 
the claim follows from Lemma 15. 1[ □ 

The above lemma and the corollary lead to the following result. 

Theorem 5.3. Consider the process satisfying (13.11) and the associated discounted cost 
functional Ja{x,u) described in (15.11) . Then, for each initial point x > 0, there is an optimal 
control u* >0 such that 

Ja{x,U*) = inf Ja{x,u) = Va{x) , 

where Va{x) is the value function of the discounted cost problem defined in (12.81) . 

Proof. Fix any x > and a > 0. By Corollary l5.2[ Ja{x, u) is finite for each m > and is con- 
vex in M-variable. By (15. 4p . we have that Ja{x,u) > and hence, since lim h{u) = +oo, 

we obtain lim Ja{x,u) = +oo. Since Ja{x,u) is convex in w-variable, we can conclude that 

there is a m* > (which may depend on x) such that Ja{x,u*) = miu>o Ja{x,u). This 
completes the proof of the theorem. □ 

Remark. Notice that in contrast with the long-run average cost minimization problem, we 
cannot rule out the possibility u* = here. 

Corollary 5.4. For the special case p = 0, assume further that h{x) is constant on an 
interval [0, 6] for some S > 0. Then, for every initial point x > 0, the optimal control u* is 
strictly positive. 

Proof. It follows from (15.41) that Ja{x,u) = Ja,i{x,u). The function C is increasing and 
X!^^{t) < X^2(t) for alH > and Ui > U2. Therefore Ja,i{x,Ui) < Ja,i{x,U2) for ui > U2- 
Consequently, Ja{x, 0) > Ja{x, u) for all u > 0. Hence, we can find an optimal control u* > 0, 
and the proof of the corollary is complete. □ 
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6 Finite horizon problem and Abelian limits 

In this section, we establish Abehan hmit relationships among the value functions of three 
stochastic control problems introduced in Section 12. 2[ The main result is stated in The- 
orem 16.31 below. Throughout this section, for simplicity, we assume that cr('u) = 1 in the 
model described in (12.11) . 

We begin with the existence of an optimal control for the finite horizon control problem 
introduced in Section fI72[ 

Proposition 6.1. For any fixed x > and T > 0, we have: 
(i) I{u, X, T) is finite for each u > and is continuous in u > 0. 
(a) lim /(m, X, T) = +00. 

u— >oo 

(Hi) If h and C are convex functions, then I{u,x,T) is a convex function of the variable u. 
Corollary 6.2. For any fixed a; > and T > 0, we have: 

(i) There is an optimal control u*{x,T) > such that l[u*{x,T)) = min„>o /(m, x, T). 

(a) In the case p > 0, if h and C are convex functions, then u*{x,T) is unique. 

(Hi) In the case p = 0, if h is a strictly convex function and C is a convex function, then 
u*{x, T) is unique. 

The proof of the proposition and of the corollary is a straightforward adaptation of the 
corresponding proofs given in Section [3l and therefore is omitted. 
The following theorem is the main result of this section. 

Theorem 6.3. Let satisfy (13.11) and letVo, Va{x), V{x,T) be the value functions defined 
in (12.61) . (12. Sp . and (12.91) respectively. Then the following Abelian limit relationships hold: 

lim aVaix) = lim Y-i^ — 1 = Vq. 

a^0+ T^oo T 

We prove this result in Propositions 16.51 and 16.61 below. The following technical lemma 
gathers necessary tools to establish lim aVa(.x) = Vq. 

Lemma 6.4. Let u > be given and X!^ satisfy (13.11) . Consider the cost functional Ja{x,u) 
as defined in (12. 7p . Then 



lim aJa(x,u) = liu) 



where I{u) is described in (13.211) . 



Proof. First consider lim aEij^e °'^dL'^{t)), where is as in (12. 2p . Similarly to (15. ip . 
we have 

e-"*dL^(t)) = e-"*L^(t)dt) = ^ e-'''E{L^^{t))dt, (6.1) 
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where we used Fubini's theorem to obtain the last identity. By (13. ip . 

E{L:{t))=ut + E{X^{t))-x. 

Therefore, 

poo poo poo 

aM e-'''E{Ll{t))dt = a\ e-^Hdt + e-'''E{X^{t))dt - ax 

Jo Jo Jo 

POO 

= u + a^ e-'^'E{Xl{t))dt-ax. (6.2) 
Jo 

Further, by (I32D, < E{X^{t)) < Ko{l + t^), where the constant i^o > is independent 
of t and u. Thus 

poo poo 

< aM e-'^'E{Xl{t))dt<Koa^ e-'^\l + t^)dt 
Jo Jo 

< KQ{a + G&mma{H)a^-^), (6.3) 

where Gamma(i7) := e'^t^dt = a^'^^ e'^^t^dt is the Gamma function evaluated at 
H. 

Since H e (0, 1), it follows from that lim /°° e-"*E(X^(t))rft = 0. Hence, using 

a— >0+ 

fl6.ip and (16.20 . we obtain 

\im^aE(^f e-"*rfL^(t)) = M. (6.4) 
We next consider lim aE{JI^ e^'^^C{X^{t))dt) . It follows from (I3A8D that 

E[j\~^'c{x:{t))dt)-E[j\-^'c{x:{t)) I dt) 
< E(^l^^\c{x:{t)) - c{x:{t))\dt) < oo, 

where X* is the stationary process described in (13. Sp and (13. 6p . Therefore, 

poo poo 

lim aE{ / e-°'C{X^{t))dt) = lim aE{ / e-"*C(X*(t))dt) 

POO 

= Mm a e-'''E(C{Zu))dt = E(C{Zu)) =G{u), (6.5) 

a^0+ Jq 

where G{u) is defined in Section [3731 It follows from (12.71) . (16. 4p . and (16. 5p that 

lim aJa(x, u) = hiu) + pu + Giu) = I(u), 

a-*0+ 

where I{u) is given in (I3.2ip . This completes the proof of the lemma. □ 
The next proposition contains the proof of the first part of Theorem 16.31 
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Proposition 6.5. Let X" satisfy (13.11) and Va{x) be the corresponding value function defined 
m diS]). Then 

lim aVa{x) = Vq < cxd, 
0^0+ 

where Vq is the value of the long-run average cost minimization problem given in (12.61) . 

Proof. Fix the initial point x > 0. For any -u > 0, we have aVa{x) < aJa{x,u). Hence, by 
Lemma [6.4[ hmsup aVa{x) < lim aJa{x,u) = I{u). Therefore, minimizing the right-hand 

side over m > 0, we obtain 

lim sup 0:^0,(0;) < inf I{u) = Vq. 

It remains to show the validity of the reverse inequality, namely that 

lim inf aVa{x) > inf I{u) = Vq. (6.6) 

To this end, consider a decreasing to zero sequence a„ > 0, n G N, such that 

lim inf aVa{x) = lim anVa„{x). (6.7) 

a— >0+ n^oo 

Fix any Eq > and let > 0, n G N, be a sequence such that V^„(a;) + Eq > Ja„{x,Un)- 
Then 

OinVa^ix) + anSo > anJani^^^n) > h{Un). (6.8) 

Letting n — ^ oo we obtain 

lim sup /;.(«„) < limsup a„V^„(x) < Vq. (6.9) 

n— >oo n— >oo 

Since lim h{x) = +00, this implies that Un is a bounded sequence. That is, there is M > 

such that Un G (0, M) for all ri G N. Therefore, without loss of generality we can assume 
that Un converges as n — > 00 to some Uoo G [0, M] (otherwise, we can consider a convergent 
subsequence of m„). 

Let 6 G (Moo, 00). Then, 

anJaAx,Un) > h{un) + alp / 6"""*^ (L^" (t)) c^t + «„ / e-'^"*^ [C (xj (t)) ] (it 

7o Jo 

Since E[L'^'^{t)) = E[X^"{t)) + - x > - x, we obtain 

Jo 



and hence 



/■oo 

anJaA^,Un) > K'^n) + p M„ - pa„x + «„ / e"°"*E [C (xf (t) ) ] (it. 

Jo 
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Therefore, letting n go to infinity and using (I6.5p . we obtain 

liminf a„Jc,„(a;,M„) > h{uoo) + pu^ + E\C{Zs]\. (6.10) 

n— >oo 

In particular, one can conclude that u^o > 0, because otherwise, letting 6 tend to and using 
Lemma 1X51 we would obtain that liminf„^oo C(nJa„ix, Un) = +oo which contradicts the fact 
that lim sup On Jq,„ (a:, < Vq < oo according to (16. 8p and (16.91) . 

n— >oo 

Therefore -Uoo > 0. Letting 5 in (16.101) tend to Uoo and using again the continuity of 
G{u) = E^C(^Zu)] proved in Lemma [3. 3[ we obtain 

liminf a„J„„(x,M„) > h{uoo) +pUoo + E[C( ) > Vo. (6.11) 

n^oo 

The inequalities (16.81) . (16.91) . and (16. lip combined together yield 

lim anJaAx,Un) = /(Moo) = Vq, 

n—too 

which completes the proof of the proposition in view of (16. 7p . Notice that (16. lip implies 
that Uoo is an optimal control for the long-run average cost control problem. □ 

The following proposition includes the second part of Theorem 16.31 

Proposition 6.6. Under the conditions of Theorem \6.3[ we have 

lim }^ = V„. 

T^oo T 

Proof. It follows from (12. 5p and (I2.10p that for any x > and m > we have 

I{u,x,T) 
hm sup — = I[u), 

T^oo T 

where I{u) is given in (13.210 . Therefore lim sup ^(•^'^) < lim sup ^^^^^^ = /(-u), and, 
minimizing the right-hand side over u > 0, we obtain 

lim sup ^^^'^^ < inf I{u) = Vq. (6.12) 

It remains to show that 

lim inf > K). (6.13) 

T — ^oo 1 

The proof of (I6.13P given below is quite similar to that of (16.60 . Consider a sequence of 
positive reals (Tn)neN such that lim T„ = +oo and 

n^oo 
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Fix any Eq > and for each n G N chose m„ > such that V{x,Tn) + Eq > T„). 
Then, in view of (12.101) and (16.121) . we have 

V(X, Tn) 

hmsup h{un) < hmsup — — — < Vq < +oo. 

n— »oo n— ►oo J n 

Since hm h{x) = +oo, this imphes that Un is a bounded sequence. That is, there is M > 

X— >oo 

such that Un G (0, M) for all n G N. Taking a further subsequence if necessary, we can assume 
without loss of generality that m„ converges to some Uoo G [0,M], as n — > oo. Furthermore, 
Moo > because if Uoo = 0, then by (13.201) we obtain 

Vo > lim sup }li^2lA > lim sup ^("'^'f'^") > e (CiZs)) for any S > 0. 

n— >oo J-n n— >oo J-n 

This is impossible in view of part (iii) of Lemma [3.3[ Therefore Uoo > 0. Let vi,V2 be any 
numbers such that < f i < Moo < V2- Then, by (I2.10p and an argument similar to the 
derivation of (16.101) . we have 

/•Tn 

V{x,Tn)+Eo > I{Un,X,Tn) > h{vi)Tn + E {LI' (Tn)) + / E [C {X^^ (t))] dt, 

Jo 

for n large enough. Using Lemma 13.11 along with (13.201) we deduce that 

liminf > h{v,)+pv, + E{C{Z,,)). 

Since vi and V2 are arbitrary numbers satisfying the above inequality constraints and h{u) 
and G{u) = E(^C{Zu)) are continuous functions, this implies 

Vo > hminf > /i(moo) +PMoo + E{C{Z^J) = J(moo) > Vq. 

The proof of ( I6.13p . and hence the proof of the proposition is complete. In fact, the above 
inequality also shows that Uoo is an optimal control for the long-run average cost control 
problem. □ 

Propositions 16.51 and 16.61 combined together yield Theorem 16.31 

Remark. The proof of Propositions 1 6. 51 and \6.6[ imply the following results: 

1. Let (a„)„>o he a sequence of positive numbers converging to zero and letUn he an E-optimal 
control for Va„{x) in (12.81) . Then the sequence {un)n>o is hounded and any limit point of 
{un)n>o is an optimal control for Vo defined in (12.61) . 

2. Let {Tn)n>o he a sequence of positive numhers such that lim T„ = +oo. // he an e- 
optimal control for V(x,T„) in (12. 9p . then the sequence {un)n>o is hounded and any limit 
point of {un)n>o is an optimal control for Vo defined in (12. 6p . 
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